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of realizing the distributed cfaedq>ointing for processes 
wtddi carry out the inter-prooess communications, while 
shortening a stopping time the processes due to the 
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tions of each process are stopped while cOTtinuing the 
nonnal processing of each process first, and a cfaec^xdnting 
processing for each process is executed when the inter- 
process communications of all processes arc stopped. 
Alternatively, the intcr-conqniter type inter-process ccaimm- 
nicatians of eadi process are stopped while continuing the 
intra^on^kater type inter-pcooess communications and the 
normal processing of each process first, and a checkpointing 
processing for each process is executed while stof^^ the 
intra-computcr type inter-process communications of each 
process when the inter-oonqKiter type inter-process coznmu- 
nications of aU processes are stofiped. 
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METHOD FOR CHECKPOINTING IN 
COMPUTER SYSTEM UNDER 
DISTRIBUTED PROCESSING 
ENVIRONMENT 

BACKGROUND OF THE INVENTLON 

1. Fidd of tbe InveotioD 

The present inventioo relates to a melfaod for checkpoint- 
iDg for a plurality of processes qjerating under a distributed 
processing environment while carrying out inta-process 
communications, and a conQHita system utilizing the check- 
pointing method. 

2. Description of the Background Art 

In general, a con^Hitex- system is provided with various 
recovery functions in order to increase a relialHHty of 
program execution. As one sudi recoveiy function, the 
cfaedqxdnttetart scheme is coQveati<maUy known. In this 
cfaedqxiini restart scheme, an information necessary for 
re-execution from a state in a middle of executioD of a 
program is legulady stored. A point for storing this infor- 
mation is called cbcc]q>oint, and an act of storing this 
information is called checkpointing. When a program execu- 
tion is intenupted by an oocurcenoe of a trouble, a state of 
that intetnq>ted process is tolled back to a checkpoint 
immediately before an oocunenoe of a trouble, and this 
process is re^ecuted from that checkpoint 

In a system io whidi one foocess operates independently, 
the checkpointing for intermediate states of that process 
alone is suffidrat, but in a distribatcd system in ^ch a 
plurality of processes operate in parallel while canying oat 
inter-process conmmnications, the ched^xnnting for a 
single process alone is insufiSdeoL Nanoely, there is a need 
for the checkpcanting for a plurality of processes which are 
related with each other by the intcr-process communications 
so that these processes can be re-executed without a con- 
tradiction. In Ae following, a cfaeclpoint far each process is 
refcxTcd to as a local chedqpcont, and a set of local dkeck- 
points ootiespoQding to the related processes is refdred to as 
a distributed checkpoint 

Per the inter-process communications among processes 
under the distributed environment, a message passing and a 
data exchange using shared memoiy or file sharing are 
available. Ihe noessage passing is a means for canying out 
the data exchange by synchronizing a message transmission 
side aiul a message reception side. The shared memoiy is a 
memory shared among a i^urality of processes, which can be 
directly read and written fay each process, where a result cf 
writing can be seen by any of these processes. A file is 
accessible from a phiraUty of processes, and an infonnatlon 
exdiange can be realized when one process writes an 
information into the file, and the other process reads out that 
information from that file. The message passing is a syn- 
dironoos later-process oonmumication, whereas the infor- 
mation exchange using shared mcmoiry or file jghjmng i$ an 
asynchronous inter-process oonmumlcarion. 

In addition, tlie shared memocy is used for inter-process 
communications within the same con^iuter, whereas the 
message passing can be used for the inter-prooess commu- 
nications between different oon9)uters, as well as for the 
intcr-process communications within the same oGmputex. 
The file can be shared among processes within the same 
conqxitcr, and tiie file can also be shared among processes 
0^ different cooqmtcrs by utilizing a networic file system, so 
that the file can be uttUzed for the intcr-process communi- 
cations between different compters. Note however that 
there is a type of file system which does not guarantee that 
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an information written by one process is going to be valid for 
the other processes immediately. In the following, the inter- 
process communication between the computers will be 
refeired to as an inter-oon^cr IPC, while the intcr-process 
5 communication within the same computer will be referred to 
as an intra-computer IPC. 

The information which must be stored at a time of 
distributed checkpointing includes the following. 

(a) A content of an address space of a process, and a 
content of a hardware roister in an execution state. 

(b) A state of file in^t/outpuL 

(c) A state of a communication path used in the message 
passing. 

15 (d) A State of a shared mcmoiy. 

Of these, (a) is a process local state, while (c) and (d) are 
states that can be affected by the other processes, (b) is a 
process local state in a case (tf not using a file sharing wi& 
the other processes, but is a state that can be affected by the 

20 other processes in a case of using a file sharing with tbe other 
processes. Namely, for a (ffooess whidi does not carry out 
the inter-process communication wifii the other process, it 
suffices to store states of (a) and (b). For a method for staring 
states of (a) and (b), see M. litzhow and M. S<domon: 

25 **Supparting Checkpointing and Process Migratioa Outside 
The Unix KemeT, Proc, of USENK Association Winter 
Conference 1992, pp. 283-290 (Jan. 1992). 

Noonally, an address space of aprocess has a text region 
for storing execution codes, a data region for storing data fcH- 

30 execution, and a stack region for storing transicot variables 
during execution. 

Id the method disclosed in the above noted reference, at 
a time of local checkpointing, a ocntent of an address space 
is stored after a normal processing is stc^jped. hi addition, in 

^5 order to stose an execution position at a tunc of a noonal 
processing stc^, a content of a hardware register is stored. As 
for tbe file iqwit^outpot, an opened file path and a seek 
pointer for indicating a fiJe operation position at a tin^ of a 
oonnal processing step are stCHcd. At a time of resuming 

40 execution from a local checkpoint, (1) an address space is 
recovered, (2) a file is le-cpoied aid a seek pointer is set, 
and (3) a stored hardware register content is set, and then an 
execution is resumed. 
On tiie odicr hand, for (b), (c) and (d), states of a plurality 

45 ofprocesses which cany out the distributed processing must 
be stared in such a manner which maintflins the consistency. 
FIGS. lA, IB and IC show some exanq>les of the distributed 
checkpointing for processes which caxiy out the inter- 
process communicatioas by die message passing. 

50 FIGS. lA, IB and IC show examples of three types of 
distributed chedqpoint timings CHI, CH2, and CH3 in a 
case where a processing is canied out while each one of 
three processes pi, p2 and p3 carries out the message 
passhig. Each CEf these timings CHI, CB2 and CH3 is a 

55 synchronous checlqpointing timing, whidi is given by a 
management process which commands the checkpointing to 
the processes pi, p2 and p3. Also, in FIGS. lA, IB, and IC, 
a symbol m indicates a message, and two numerals suffixed 
to this symbol m Indicate a message transmission side 

60 process numher and a message reception side process num- 
ber respectively. 

In FIG. lA, at CHI, die processes pi, p2 and p3 are 
dieckpcdnting the local checkpoints chll, chl2 and chl3 
respectively, but for a message m32, despite of the feet that 

65 the process f3 Is still in a state of not transmitting this 
message at cfal3, the process p2 Is in a state of already 
received this message at chl2. For this reason, when a 
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trouble occurs in any one process and the processes are to be exchange using shared memory or file sharing, while sbort- 

ct-cxecuted by rolling back to CHI, a oontradictioo regards enlng a stopping time of the processes due to the distributed 

ing the message m32 arises. Similarly, for CH3 of FIG. IC, chcclqpolnting. 

the coQtradiction regarding the message m23 arises. Accordmg to one aspect of the present invention there is 
Id contrast to these, at CH2 of HG. IB, there is no 5 provided a method for checkpointing for a plurality of 
contradiction for each message, so that the roll back and fbt i^ocesses operating under a distributed iaxx:essing environ- 
re-cxecution can be realized correctly. ment while carrying out inter-process communications, ttie 
As a scheme for guaranteeing the consistency of the method conqxising the steps of: stopping the inter-process 
distributed checlipoint, there is a scheme disclosed in K. communications of each process while continuing a normal 
Mani Chandy and L. Lamport: ^'Distributed Snapshots: lO processing of eadi process; and executing a cfaedtpointing 
Determining Global States of Distributed Systems**, ACM processing for each process, when the inter-process conk- 
Trans, of Con^Hiter Systems, VoL 3 , No. 1 , pp. 63-75 (Feb. munications of all processes are stopped at the stopping step. 
1985). This sdieme deals with the nicssage passing as the According to another aspect of the present invention there 
intcr-process co mmn iri c a t ion, similariy as the examples j»ovided a mc&od for checkpointing for a phirality of 
described above, and defines the consistent distributed 15 processes cpeiating under a distributed processing enviroo- 
chedqpoint as "a state without a message which is ncc yrt carrying out inter-process coimmmlcations, die 
transmitted and afa-eady recdved". The message m32 of FIG. njethod conning the steps of: commanding each process 
lA which is distributed cfaedq>ointing at a timing of CHI is ^ ^ ^ intcr-process communications of each process 
a message which is not yet transmitted and alrea<ty received, continuing a normal paocessing of each process; and 
so that CHI is an inconsistent timing. 20 cwnmandlng each j^ocess to execute a chedqxanting pn^ 
Also, in this sdicme, a message re-transmission is carried ccssing for each process, when a stopping of the im«x- 
out in order to maintain the consistency. For example, the process communications of all processes is confirmed. 
messagem23ofnG.lCisgoingtobeamessagewhichis AaxHding to anotha aspect of the trocnt invention tbae 
already transmitted and not yet received when the processes . ^ aiKmia w ^ 
Sw^Sc^ and^e consistent trouhleVeoovery 25 ^ ^"^^ * "S^^^f^ 
^Tc^ oatbyiht re-transmission of this mess^ processes opiating unto a distributed processing envtrcn- 

Also, in tto scheme, a consistent state is realized by mdhodooinprisingthc stqs oft Soi^atowhpro^^ 

8ei«Wamcssage«dkdntttobetwecnthepr«eMes.By T^t^ comiminicatwns of each process whflecoa- 

^caroroittbenie.s.gep«SI^brc«xi.S^ ^^"^IS^^S^rffSSIlSr'lSf 

tfcT^^cWr^ at each process en execution of a checkpointing processing 

stmewblcfarMint^theccmsisteocy. fcreach^process.imonrec^ 

However, this scheme presupposes the syndnonousmtcr- iw«u« utwu*w«Ym|5 o *^5**4^*u«*i5 

irocess communication such as the message passing, and Aojording to another aipcrt 

there is a problem that it is not qipdicabte to an asyndironous 35 ^ provided a mediod fOT cfaeckpointuig for a pluralily of 

intcr-process commnnlcation such as an information processes on conqwlers, the processes being operating under 

exchange using shared memory or file sharing. » distributed processing cnvironmcK while carrying out 

On the other hand, apart from the above sdicme i»tposed inta-process communications indudingintcr-conqjutcr type 

by Mani Chandy ct aL, J. S. Plank and K. U: *ic±p: A inter-process commnnications to be carried oat among the 

ConsistemOied^wintn: for MuWoomputexs^ IEEE 40 conqjutcrs and intia-con^wter type intex-processomi^^ 

lei & Distributed Ibdmology System Application, VoL 2, cations to be carried out within one conqniter, ttie method 

No. 2, pp. 62-67 (1994) proposes a sdicmc for local comprising the st^ of: stppping tbt intcr-oonqjuter type 

diwipoanring by >-^«hHdiing a syncinonizalion using the intcr-process communications ctf each process while coo- 

two-i^e commit protocol. In this scheme, an the mumally tinuing the intra-conqxiter type inter-procfss communicar 

related processes arc stopped at the first phase and a state 45 tions and a namal processing of each process; and cxwu^ 

with no message state is produced, and tiien after the local "^8 » checkpointing processing for cadi process ^*flc 

(iie(±pGdnring for all the processes is carried out, the pro- stopping the intra-<janqjuter type inter-pnxess communica- 

cessing of all the processes is resumed at the second phase. of «*cb process, when the inter-conyuter type inter- 

Kamdy, the distributed die(±pointing is to be carried out process mmmiinicntinns <rf all processes arc stopped at the 

in a state at CH2 shown hi FIG. IB. This scheme is 50 stopping ^ep- 

^jpUcable to bodi the message passing and the infonnation Aoooixling to another aspect of the present invention there 

exchange using shared mcmoiy cr file sharing by prohibiting is provided a method for chedqwinting far a plurality of 

an access to a shared memory or a file at the first phase, but processes on computers, the processes being (^>erating under 

the processing of an die numuUy related processes are to be a distributed processing environment while carrying out 

stopped at the first phase, so that there is a problem that the 55 inter-process oommimicationsinchidinginter-^oiqMtcr type 

process stopping time becomes very long. In ofiier words, in inter-prooess communications to be carried out among the 

this scheme, the chedipointing of the local dieckpoints is computers and intra-compotcr type inter-process communi- 

made after the processing of aU the related processes is cations to be carried out within one computer, the method 
stopped in synchronization, so tliat there is a problem tiiat conqoising the steps of: ooramanding each process to stop 
the normal processing is oonyletely stopped during the 60 the inter-conqyuter type inter-process communications of 

distributed checkpointing. each process while continuing the intra-conqwitcr type intcr- 

„ process comnumications and a normal processing of each 

SUMMARY OFTHBINVEOTION process; and coinmandhig cadi process to execute a diecfc- 
B is therefore an object of the present invention to provide pointing processing for each process while stopping the 
a sdicme for checkpointing and a conyuter system capable 65 intca-oomputer type inter-prooess communications of each 
of realizing the distributed chedqKiinting for fffocesses process, when a stopping of the intcr-coiEpiter type inter- 
wMdi carry out the message passing or the Informatioo process ccHxuuunlcations of aU processes is confirmed. 
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Acoonliiig to another aspect of ttic present invention tbcie municatioos of each process means upon receiving the 
is provided a method for checlqpointing for a plurality of cbedspointing command. 

on coniputers,theprocesscs being opcraa^ unda other features and advantages of the present invention 
a As^ted processing env^onment while canying out ^ become apparent ftomttefoUowing description taken 

mto-processcommnmcadonsinctadinginte^ j In conjunction^ the accompanying Swings 

mter-proccss communications to be earned out among the b 

oomputcrs and intra-computer type inler-proccss communi- BRIEF DESCRIPTION OF THE DRAWINGS 

cations to be carried out within one con^wtcr, tiie m^Kxl piGS, lA, IB and IC arc diagrams showing ocanmlcs of 

oompnsing the steps of: stopping at each process (he Inter- conventional distributed checlqjoim timings in a case of 

computer type inter-process c omTnuni ra t ions of each pro- iq using the message passing. 

cess while continuing the intraHcomputcr ty^ in^-proccss piG. 2 is a Wock diagram showing a functional configu- 

commum<^om and a normal processing of eadi process, ^ distribut^^tcm to Ihich tbe^^S 

upon recetviBg an uto^uter type interijoccss com. eheckpoinUng scheme of me first embodimem\<xS1o 

mumcation stop command; and starting at each process an uie^enl ilvcntion is aptdied. 

execution of a chcclqxjmting processing for each process 13 rio » • a ^^I/Z ^ ^ ^ l . . 

while stepping the in£a-co4ito type inter-proce^ com- ^^'^^ flow chart of the oparation by t^j±eckpoint 

munications of cadi process, upon nieiving Tcheckpolnt. <=omttmid process in the distributed system of HG. 2. 

ing command. ^ ^ ^ flow chart of the operation by the oomnmni- 

Acoordingtoanotheraspectof thepreseDtinvcntionthere ^ distributed system of HG. X 
is provided a computer system, con^Hising: a phirality of 20 5 is a flow ciiart ci the opcratiwi by the local 
process means operating under a distributed processing checkpoiitfing unit in the distributed system of FIG. 2. 
environment while carrying out inter-process commnnica- ^ is a diagram showing a relationship of (he opera- 
tions; and chedqpolndng command means for issuing an ^^^^ aniong two processes and the checkpoint command 
inter-process communicaticm stop command commanding a process in the distdbuted system of FIG. 2. 
stopping of the inter-process comnuinications of each pro- 2s ^ ^ block diagram showing a fonctioaal oonfigu- 
cess meanSf and issuing a checkpointing conunand com- ration of a distributed system to which the distributed 
manding an execution of a checbpointing processing for checkpointing scheme of the second embodiment according 
each process means when responses to the inter-process ^ the present invention is applied, 
cammunication stop commands from all process means ate FIG. 8 is a flow chart of the operati<m by the local 
received; wherein each process means 8t<^ the inter- 30 <iieckpointing unit in the distributed system cf FIG. 2. 
process cofnmnnfcations of each process means while coo- FKj. 9 is a flow chart of the opctadon by the page 
tinuing a normal processing of each process means i^n management unit in the distributed system of FIG. 2. 
receiving the hiter-process communication stop command PlG. 10 is a diagram showing a rdationsh^ of the 
and returns a lespcuise to the inter-process ccmmunicatioD operations among two processes and the cfaedpoint comr 
stop command i^ien the inter-psooess commnnications of 35 mand process in the distributed system of FIG. 7. 
each process means are stopped, and starts an execution cf piQ. U is a Uock diagram showing a functicxial configu- 
ihe checkpointing processing for each process means ^>on ration of a distributed system to which the distributed 

receiving the checkpointing command. rhfy^inring grhwrm nf thr^ thir A ^^mhoHi'm^w t aronMIng tft 

According to another aspect of the present inventtOQ there the present invention is qifdiod. 

is provided a oonqmter system, conqffising: a plurality of 40 fjg. 12 is a flow chart of the operation by the page 

process means provided on conqnitas, the process means munflgMnMit unit in the distributed system of FIG. 11 

bdijg operating under a disto^ FIG. U is a flow chart of the operation for the local 

while canymg out inter-process communications mduding checkpointing processing by the locald»eckpointing unit in 

mter-con?iutcr type intcr-i»occ8s communications to be the distributed system cf PIG. 11. 

c«riedom among the compute a^ 45 nOS. 14A and 14B are timing charts for the checkpoint- 

S.^IS^n^'Z^i'^^ to be ojined om with^ one ^ ^ ^ , ofT^onventional sc^^ a 

wmputcr,and<Jeckpoi^ J^^^ embodiment of the present invention, 

inter-cooqwter type hitcr-pioccss communication stop com- rem^ wuLPwumK^m «i luc hh^cul mvoiuuii, 

mand co mmandin g a stof^slng of the inter-con^jutcr type : w • * ^ . _^ 

inter-process communiaSirof cadi prooeTS^ aid 50 ™* ^ ' ^f^^™"! 

issuing a dieclpoindng command con^manding an eiecu- ^?°f J ^^^"T^J^JZ ^^-^ 

tionofadiw^olntin^^ checkpointing sdiei^ o^^ 

and a stopping of totta:^tcr type inta-process ^tV^^ ^''^f'^ ^ "f^?*' 

communications of each process means when icfiponses to ™- ^ ^ diagram showhig a relationship of the 

the inta-K»mputer type inter-process cammunication stra 55 op^tions among the system management unit, the process 

commands from all process means are received; wherein management units, and the processes in the distributed 

each process means stops die Inter-conqMitcr type inter- system of TO. 15, 

process communications of each process rtiMnn while con- 17 is a flow chart of the operation by the system 

tinuing the intra-conqato type Inta-proccss communica. management unit in the distributed system of nG. 15. 

tions and a nomial processing of each process means upon 60 ^G, 18 is a flow chart of the operation for the process 

receiving the intcr-compatcr type inter-process oommunica- management unit in the distributed system of FIG. 15. 

tion stop oHnmand and returns a response to the inter- FIGS. 19A and 19B are diagrams showing cxcii4)lary 

cos^Niter type inter-prooess cooununicatioa stop command timings in a case of delayed message transndssioQ in the 

when the inter-conqNiter type inter-process communications fourth embodiment of the present invention, 

of each process means are stopped, and starts an execution 65 FIG. 20 is a diagram showing an exen^laiy timing for a 

cf the di rrkpm'ntin g processing for each process means processing to deal with a case of delayed message transmis- 

while stof^ung the intra-coiiq;»uter type inter-process com- sion in the fourth embodin^nt of the present invention. 
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no. 21 is a block diagram showing a functional configu- process M, oommunication control unit 11, and local check- 
ration of a distributed system to which the distributed pointing unit 12 at a time of distributed dicckpointing will 
checkpointing scheme of the fifth embodiment according to be described. FIG 3 shows a flow chart of the operation by 
the present invention is applied the dieckpoint command process M, FIG. 4 shows a flow 

FIG 22 is a flow chart of the operation by the commu- 5 chart of the operation by the comnmnicatlon control unit 11, 

nication control unit in the distributed system of FIG. 21. and HG. 5 shows a flow chart of the opemion by the local 

FIG. 23 is a flow chart of the operation for the local <*eck|X)inting unit 12. 

chcckpdnting unit in the distributed system of ¥IG. 21. At a time of distributed checkpointing, the checkpoint 

FIG. 24 is a flow chart of the operation for the intra- „ command proofs M notifies the ooi™ 

cormt^ unit in thTS>uted system of ^ commuiu^tionconlrol unit 11 of each one of 

FIG21 ^^^^^^ ^ the related processes PI to Pn, so as to request the stopping 

rr^^o* , ^ *i 3 -t.-** of conmiunication (stq> SU). 

FIGS. 25A and 25B are timing charts for the checipomt- ' , « ^ .x. 

ing processing in a case ofTfirst to fourth embodiments ^ oonmmnication control unit U of eadi process is 

^fd a case of me fifth embodiment of the present Invention, ^wa^g for tiie communication coatro^ command (step 

ax^nks ^ conamunicatlon stop coimnand is 

icspecxiveiy. received (stq> S22 YES), the intcr-process commnm'raticms 

DET AILED DESOUPnON OFTHE are stopped (step S23). Then, a response is rcturiKsd to flie 

PREFERRED EMBODIMENTS checkpoint command process M, so as to notify that the 

Refaring now to FIG. 2 to HG. 6. the first embodiment inter-prooess conmmnications are sUjippcd (step S24). 

of a scheme for checkpointing according to the present ^ The diedq>oint command {socess M is awaiting far 

inventicQ will be described in detail respemses ficom all the conuminication control uniu 11 (step 

FKj. 2 shows a functional configuration of a distributed S12), and when all the responses are received, the local 

system to which the distribnted cfaec^>ointing scheme of checlqpointing request is issued to the local cfaedq point in g 

this first embodiment is qipHed. This distritHited system unit 12 of each process (stq> S13). 

conqsises a jdurality of con^Hiters, which ate operating in ^ The local checkpointing unit 12 is awaiting for Qie local 

Tniinifti cooperation. On these conqxiters, a plurality of checkpointing request (Step S31), and wlien the local cfaeck- 

mntimity related processes PI to Pn are executed. The pointing request is received, the nomial prooessiiig of the 

checkpointing timings fcr these processes PI to Pn are given corresponding process is stopped (st^ S32), and the local 

by a diecJqpoint command process MNamdy, the synchro- checkpointing is started (8tq> S33), when the local check- 
nous chec^>ouit as described above is used here. ^ pointing is finished, the local checkpointing unit 12 resumes 

The checkpoint command process M is executed on one the noonal processing (Stq> S34), whflc requesting the 

oonqjutcr in this distributed system, and realized as an resuming of oommunication to the c ommuni r nr i<Hi control 

operating system, or as a process indq)e]ident &om the unit U (sUp S35), 

operating system. The-^chedqKunt oommaud process M When die communication resume request is received (step 

commands the local checkpointing timings to the processes S25 YESX the communication control unit U resumes the 

PI to Pn A rVy^ i^^'ntf "g «^maf»t k iitxacA rrgularty at an intcT-process communicati<His (^ S26). 

inters of several seconds, for example;. In addidoo, die local checkpointing unit 12 returns a 

The dxeckpolnting protocol is carried out by utilizing the respcmse to die checkpoint conmiand process M, so as to 

two^jhase scheme. At the first i^iase, the inter-process ^ notify the finish of the local c h ec kpoint i n g (step S36). 

oommunication stop command is issued to each of the xhe checkpoint command process M is awaiting for 

processes PI to Pn. When the stcq)(dng of all die inter- reepooses from all the local checlqyrinting units 12 (step 

process communications is confirnKyl, at the second phase, S14), and when all the responses ate received, the di^rib- 

the local cfaec]qx>inting is commanded to eadb of the pro- uted checkpointing Is completed, 

cesses PI to Pn. 45 lb this manner, in this first embodiment, the processing to 

The processes PI to Pn are operating while carrying out g^^p Ihc cxecutiDn of the inter-process oommunlcatiOiDS of 

the intcr-process communications. Each of these processes the processes PI to Pn is carded out at the first phase, and 

has a communication contrc^ unit Hand a local checkpoin^ when the stc^iping of afithe inter-process oomnniiricatfons is 

ing unit 12 as functions for the checkpointing control confirmed, the normal processiog is &toppcd at the second 

The communication oontroi unit 11 is realized as a 50 phase, and the checkpointing for e&di process is started, 

communication library or a oonmuinicati<Hi manager on a Ihen, the inter-process communication and the normal pro- 

mioo-kemd, for example, and carries out the stopping and oessing are resumed at each process for ^cfa the ched:- 

tfae resiuning of the inter-prooess comouinications. In this pointii^ is coa^leCed. In this case, only the inter^ocess 

case, the stopping of the inter-prooess communications is ccHumunications are stopped during a period before the 

casned out in response to a communication stop command 55 cfaed^inting starts. 

from the rh^^^T^"* pntnTnand pmoeftfi and theresuming por this reason, each process can carry out the nonnal 

of the inter-process commnnirations is carried out in processing up to a st^ for carrying out the inter-process 

re^nse to a communication resume command firom die communication, and is set in a waiting state for the first time 

load Gfaeckpointing unit IZ when it reached to a step for carrying cut the inter-process 

The local checkpointing unit 12 casdes out cfaedq;Kiinting 6a oommimication. Also, in a case where the process is oper- 

ofa local dieckpointof a corresponding process in response ating under the multi-tiiread environment, even when a 

to a chedqMinting command from tiie cbecbpoint command thread which carries out the inter-process communication is 

jrocess M. When the checkpointing is finished, die local set in a waiting state, it is still possible for the other threads 

diedq[>ointing unit 12 issues a oommunication resume com- wtiicfa are not related to the inter-prooess communication to 

mawi to the conmiunication control unit II. 65 continue <^>crating. Consequently, tt is possible to shorten 

Now, with references to the flow charts of FIG. 3 to FIG. the (^>eration stopping thne for the processes as a whole, 

5, a scries of operations by these checkpdnt command compared with a conventional scheme of Flank ct aL, in 
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which the cfaec^pdnting is started after aU the operations of control unit 11 cany out the operations similar to those in the 
each process are stopped, and the resuming of the normal first cmbodimcot described above, 

^ocesslng is commanded to each process only when the The page managonent unit 13 sets the address space in a 

chedqpointmg is finished for aU the processes. writing prohibited state by using the function of the opcx- 

FIQ. 6 shows a relationship of the (^)erations among two ^ ating system (OS) at a start of the processing of the corre- 

processes PI and P2 and the cfaeclqxunt command process sponding process (st^ SSI), and an event is awaited (step 

M in this first embodiment S52). When a writing with respect to a certain page is carried 

In FIG. 6, "com stop** is the communication stop com- during the nonnal processing, a page protection viola- 

mand which is issued from the chedqxnnt command process ^on is caused by means of the OS function. When the page 

M at the first phase of the distributed checkpointing management unit 13 detects this page protection violation 

processing, and "^AGC* is a response to "^coo stop** from SS4 YES), tiie page management unit 13 records an 

each process. Also» *'CH create" is tiic local checkpointing address of that page in the dirty page list, and sets this page 

start command wldch is issued from the chec^xunt com- tn a writing permitted state so as topomit the writing by die 

mand process M at the second phase of the distributed D<smal fsooessing (st^ S55). 

checkpointing processing. Also, ^en the dirty page list request Is issued from the 
In FIG. 6, even when the transmission request for the iocal checkpdnting unit 12 (step S53 YES), the page man- 
message ml from the process PI to the process P2 occurs at agement unit D sets the address space in a writing prohib- 
a timing A, the transmission of this message ml is not ited state (step 556)^ and returns the content of the dirty page 
carried out, and after die checkpointing processing is com- ^ clearing ibc dirty page list (step SS7). 
pleted and the noEimal processing and the commnnication are When the local checkpointing request is received from the 
resumed^ the transmission of this message ml is carded out cfaeclqpoint command process M (step S41), the local check- 
at a timing B. pointing unit 12 stops the normal jsocessing of die process 
Consequently^ the commnnication of the processes PI and 542). Then, tbt local checkpointing unit 12 requests 
P2 between the respecrivc local checkpoints^ up timings ch 25 ^ list to the page management unit 13 and 
of these processes PI and F2 is pnivented, and it becomes obtains the dirty page list (step S43). After that, the local 
possible to carry out the distributed chec^Exunting in a state checlqtointing unit 12 starts the processing to store only the 
which Tnmntai'niii tbc consistency. pages rcoosdcd in the dirty page list as die local checkpoint 
Also, as described above, even ^n "con stop" is issued, infOTuation (st^ S44). and in parallel to die execution of die 
die jHocesses PI and P2 can cany out the processing odicr 30 iocal che ckpoint i ng IHoccssing (step S45), the normal pro- 
dian the interprocess communication, so that a time fcr cessing is resumed (st^ S4«) and die resuming of commu- 
stopping die DOTmal processing entirely can be shcitencd requested to the ccsnmanication control unit 11 
considerably. <^ S47). 

Rcfaring now to HG. 7 to PIG. 10, the second cmbodi- after die end of die local checkpointing is awaited 

mcntofaschanefordieckpoinling according to tiiecrosent 35 (step S48), at a point where die local checkpointing carried 

invention will be dcsoibed in detaiL ^ ^ parallel to die normal processing is finished, the local 

— ^ - . * ^ I ^ £ j...^ . J checkpointing unit 12 returns a response to the cfaedmoint 

FIG. 7 shows a functional confignration of a distributed ITZI^rTZ^^. w c?>ia\ *t~ 

system to which die distributed checkpointing sdicme of command process M (step S49). 

diis second embodiment is applied. In diis distributed ™ procedure, a penod for which die normal pro- 

systcmi, each one cf die processes PI to Pn has a page ^ ccssing is stopped is only since die dirty page list is obtain^ 

management unit 13 provided in addition to die ocamnuni- ^ local chc<Apointing is started, so diat die stopping 

cad<m control unit 11 and die local rhftpirpn»n*^>,g unit 12, ^ ^ intcr-proccss c omm«nfmti <ys and die other 

anddielocalcbec^>ointingunit 12i8madetobecq>ahLeof normal processing can be shortened in tiiis second embodi- 

execntiog the local checkpointing processing in pazallel to ''^^ 

the normal processing. ^ FIG. 10 shows aieladonshlp of the operadons amoQg two 

The page n».ni»gr^nf unit 13 has a flmcti<a to detea a procwses PI and P2 and die checkpoint command process 

page in die mcnKxy address space to which die writing is M in tids second embodiment 

carried out by the nomial processing of the conesponding ^ ^ ^ FIG. 10, die stopping time is shortened 

process, and is provided widi a dirty p^e list foriecordlng coznpared widi a case of the first embodiment shown in FIG. 

addresses of die pages (dirty pages) that have been rewritten 6, and the delay of die inter-process ccanmunicadon is also 

since die previous checkpoint up until die present dme. reducel 

The local checkpointing unit 12 stoaes only die pages RcfeningnowtoFIO. lltoHGS. 14Aandl4B,diediird 

recorded in the dirty page list as the local cfaec]q>oint embodiment of a scheme for checkpointing acocHding to die 

informadon by using a function for local checkpointing, in 55 Present invention will be described in detail, 

parallel to die normal processing. By means of dds» the local In the second embodirneiit, when the writing is carried out 

checkpointing time can be shcatened, and the stopping time by die ncamal processing while the local checkpointing 

of the nonnal prooessiiig and the inter-process comrmmica- processing is carried out in parallel to the normal processing 

dons due to the distributed checkpointing can also be and the dhty page to which this writing is canied out is not 

shortened. ^ storod in the checkpoint, die oonnal processing is stopped 

Now, with references to the flow charts of FIO. SandFIG. ^ ^ stored. In this regard, in tills third 

9, a procedure for the distributed checkpointing in this embodiment, die local chedqwinting unit 12 has an addi- 

seoond embodiment will be described. Here, HO. 8 shows ^onsl function to store an unstorod dnty page widi priority, 

a flow diart of the operation by the local checkpointing unit when the writing request to die unstored dirty page is issued 

12, and FIG. 9 shows a flow chart of the operation die 65 by tlie normal processing. 

page management unit 13. In diis second embodiment, die FIG. 11 shows a functional configuration of a distributed 

cbcckpoAnt command process M and the communication system to which the distributed checkpointing scheme of 
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this thirtJ embodiment is applied. In this distributed systan, out the checkpointing in parallel to the noansl processing, 

each one of the pioccsses PI to Pn has a priority storing the stopping time of the ncsmal processing and the inter- 

queue 14 provided in addition to the communication control process communication is shortened further in this third 

unit the local checlipointing unit 12, and the page embodiment 

management unit 13» what the address of the page to be 5 Referring now to FIG. 15 to HG. 20, the fourth embodi- 

stored with priority is to be queued in this priority staring ment of a scheme for checkpointing according to the present 

queue 14. invention will be described in detail 

Now, with references to the flow charts of FIG. 12 and FIG. 15 shows a functional configuration a distributed 
FIG. 13, a procedure fof the distributed chedqpointing in this system to which the distributed checkpointing scheme of 
third embodiment will be described. Here, FIG. 12 shows a i^ this fourth embodiment is applied. In FIG. 15, a plurality of 
flow diart of ttie operation by the page management unit 13, conq)uters CI to Cn are connected to the network, and 
and FIG. 13 shows a flow dbaxi of the conation for the local process management units FMl, FM2, etc. for managing 
checkpointing processing by the local dieckpointing unit 12 processes on the respective con^tcrs are provided on these 
which cocrespottds to the step S45 of FIG. 8. In this third computers CI to Cn. Also, each process has tiie cormmuii- 
embodiment, the chec^int command process M and the cation control unit 11 and the local chedqpointing unit 12 
communication control unit 11 carry out the operations similarly as in the first embodiment In addition, one oonir 
similar to those in the first embodiment described above. puter Cn has a system management unit SM for issuing a 
The page management unit 13 sets the address space in a checkpointing request to cadi of the process management 
writing prohibited state by using the function of the oper- units PMl, PM2, etc. on these oon^wtus. 
ating system (OS) at a start of the processing of the cone- ^ FIG. 16 shows a flow of messages for the checkpointing 
sponding process (step S61), and an event is awaited (step in this system of FIG. 15. Here, the entire protocol com- 
352). prises two phases, where the inter-process communications 
When the dirty page list request is issued tan the local are sU^ed at the first phase, and the local dieckpointing is 
diectoointing unit U (step S63 YES), the page managcnaent caxricd out at the second phase. Also, HG. 17 shows a flow 
unit 13 sets the address space hi a writing prohibited state ^ chart of the operation by the system m anage m ent unit SM, 
{step S«9), and returns the content ot the dirty page list while vAalt HG. 18 shows a flow diait for die operation by eadi 
dearins the dirty page list (st«> S70), similarty as in the process management unit VhA (PMl, PM2, etc). In this 
second embodinint f«>«rth embodiment, flic comnnmication concrol unit U and 
Also v/hea a writing with respect to a certain page is ^le local diedqxHnting unit 12 cairy out ^ ^paalions 
c.^S'oS^^«SiSng,apa^ ^ similar to those in the first emb^t descr^^ 
vioialion is caused by means of flic OS function. When flic Tbt system management umt SM requests flie first phase 
page management unit 13 detects this page inotection vio- oflhc dieckpointing to each process m anage ment unit PM 
lation(step S64YES), die page management unit 13 chedcs on eadi conqwtcr (step SIW). Each process management 
whettierttie local dieckpointing by die local dieckpointing 35 unit PM is awaiting for a request fixan ttie system nianage- 
onit U is in progress or not (step S«). If so. flie address d ment umt SM (step Sill), and ^n flic request for flie first 
the deeded page is recorded in flic priaity staring queue I^ase is recdved (step S112 YES), each process manage- 
(stcpS6«, and ttie storing cwnplction notice from ttw local ment unit PM requests flic stopping of tfie inter-frocess 
dicdtoodntingunitl2isawaitBd(8tepS^.Aftcrttie staring commnmcations to ttie comnmnication control umt 11 of 
conmletion notice is recdved, ttie page managemcm unit 13 ^ eadi coneaponding process inside flic same computer (s^ 
wxsds flie ddected page address in flie dirty page list, and S113). Each comnmnication oalrol unit U flicn stops flie 
sets tills page in a writing penniited state (step Sd8). On ttie interi»oces8 conamnnicaflons. and returns a rcqwnse so as 
other hand, when flic local cb«^>dnflng is not in progress to notify flic stewing <rf the interiwocess cotnmunications, 
(step S65 NO), (he steps S66 and S67 axe skipped, and flie Each process management unit FM is awaiting for a 
page management unit 13 only recoids the detected page 43 respoeose firom eadi corresponding communlcaflon control 
address In the dirty page list, and sets fliis page in a writing unit 11 {step S114), and when all die responses from all flie 
permitted state (step S68). communication control units 11 are received, eadi process 
The local checkpointing unit 12 ccanpictes flie local ma n a ge m r nt unit PM returns a response to tiie request for 
die<±poanting when ttictc is no more unstorod dirty page ttie first phase to flie system management unit SM (step 
during flie local chedqxnnting. If fliere is an unstored dirty 50 Slip- 
page (step S71 YES), iKlicflier the page address is registered The system management unit SM is awaiting for 
in die priority staring queue 14 or not is checked (step S72), responses from each process m a nageme nt unit PM (step 
and if so, that priority storing page is stored wifli priority S102), and when aU the responses from all flie process 
(step S73), and ttie storing oonqdetion is notified to flie page management units FM are recdved, flie system management 
management unit 13 (st^ S74). On the other hand, when 35 unit SM issues a request for the second phase of tfie 
there is no page information in flie priority storing queue 14 dieckpointing to each process management unit PM on each 
(step S72 NO), the page address is for fliat unstored page is conqiuter (step Sl<^). 

obtained from the ditty page Ust, and this page is stored (step Bach {nocess management unit FM is awaiting for a 

S75). request from the system management unit SM (step SUl), 

To illustiate the effect of this third embodiment, FIG, 14A 60 and wlien the request for the second phase is recdved (step 

^ows a case of flie cQDventional scheme of Flank et al, S116 YES), each process management unit PM requests the 

widUe FIG. 14B shows a case of fliis third embodiment In local chec^Kxnting to flie local chedqiolnting unit 12 of 

contrast to a case of flie conventional sdiemc shown in FIG. eadi oorre^nding process inside flie same conq;)uter {step 

14A where all the communication and normal laoccssing are S117). When ttie local dieckpohiting request is received, flie 

stci^ daring die cfaed^xiinting, it can be seen in FIG. 14B 62 local checkpointing unit 12 stops flie nonnal processing of 

that flic stopping rime of the normal processing is consid- the ooresponding process, and starts the local checkpoint- 

ffably shortened in fliis fluid embodiment Also, by carrying ing. Then, when flie local dicc^inting is finished, die local 
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chcclqpoiiiting unit 12 resumes the normal processing while FIG. 19B shows a case in which the checkpointing is 

requesting the resuming of communication to the commu- carried out at both processes A and B after the process A 

nication control unit 11. In addition, the local chedqxdnting issued the transmission system call, and the actual message 

unit 12 returns a re^nse to the process managemcat unit transmission is carried out after diat. When a trouble ocans 

FM so as to notify the end of the local checkpointing. 5 and the processes arc restarted ftom the checkpoints, the 

Eadi ^Dccss management unit EM is awaiting fw a process A completes the message transmission while ttie 

rc^nsc from each local diedqpointin process B is set in a reception state, so that they are 

and when aU the responses from aU the local checkpomtmg inconsistent checkpoints. 

units 12 are received, each process management unit PM -wk ^IT ^ * . .... 
returns a resp<Hise to the request for flie second phase to the F^G. 20 shows a {Housing for r 
system management unit SM (step SllS) At a time <^ response In the first jAasc of the 
The system management unit SM is awaiting for <*^^^i^^^^proccssAn^csatrmMotian^ 
responses from each process management unit PM (step ^n^^^^mttcd to «be jhoccss B. This transmissioo amount is 
S104), and when aU the responses from all the process notified to the process B by the request in the second phase 
management units PM are received, the cfaedmointing pro- ^ process management unit PM and the system man- 
cessing is conq>leted. agement unit SM. The process B delays the local check- 
By this procedure, it becomes possible to realize the pointing untQ data of the notified transmisston amount are 
distributed chcdqpointing while managing a plurality of received. By means of this processing, the timing as shown 
processes on the distributed system eflidcntiy. in Fia 19B can be prevented, so that it beccancs possiWe to 
Now. with references to HGS. 19Aand 19B and FK}. 2#, ^ consistent distributed checkpointing. Note that 
a procedwe for stqjping each inta-process communication ^ PIG- ^i* for a case where the processes A and Bare on the 
in this fourth embodiment wiU be described. Each process same computet When the processes A and Bare on different 
has a coimnunication stop flag, and the ctMnmunication «>»*»^ the shnilar processing is carried out l>y the 
control unit 11 sets this communication flag in response to P«<ocol shown m HG. 16. 

ifae o mnTnun?rj>tf on fitnp reqnagt frnm thf pTOffss mflnag ^ Notc also that the OS often pTOVides the message passing 

ment unit FM, or resets this commnnicatica flag in response ^ based on a reiiaUc protocol and the message passing based 

to the communication resume request from the local check- on unreliable proCocoL The former is tiie message passing 

pointing unit 12. without a lost message, whereas the latter is the message 

In a case where each pocess carries out the message passing in which a lost message could occut For tiie reliable 

passing or the file uqwt/outpat, cadi process issues a oomr I^<*oool, it suffices to carry oii the consistent checkpcanting 

munication system cafl with respect to the OS. In a case of *«»iding to the above described procedure. 

eaceguting thi'g nmmnnniriiHn n syrAm raH ^ tfw> fl nmmimi . It is alsOpOSSifaletO USC the aboVe described (ffOCedUie fCV 

cation stop flag is set, it is awaited until the communication unreliable protocol as well, but since the lost message is 

Stop flag is reset Here, for tiie message passing, only the allowed, it suffices to guarantee that there is no ^'message 

message transmission is stepped, and for the file input/ 35 which is trflo smi ttcd but not yet recetved'*. To this end, it is 

ouq;Hjt, only the file ou^ is stofiped. possible to ^ntpty stop the transmissi<ai alone instead of 

As for the shared memory, a writing into fee shared carrying out the exchange of the transmission amount as 

memory is prohibited. Here however, unlike the message described above. 

passmg and the file iiqyut/output, an access is made similariy Referring now to FIG. 21 to FIGS. 25A and 25B, the fifth 

as a nocmal memory access rather than issuing the system 40 embodiment of a scheme for checkpointing aooording to the 

calL For this reason, at a time of setting the communication present invention will be described in detaiL 

stop flag, the shared memoiy is set in a writing prohibited FIG. 21 shows a foncticHial oonfigcrati<» of a distributed 

state by tiie function provided by the OS. When a writing system to which the distributed chec^Knntliig scheme of 

into the shared memory occurs in this state, the page fault is thi^ Mth embodiment h applied. 

caused, so that if the communication stop flag is not set by 45 In the first to fourth embodiments described above, all the 

that processing routine, the writing prohibition is released inter-process ooammmications are stopped at die first phase, 

and the writing is pe r m it ted. If the communicatioa stop flag and the local cfaeclqx>inting is carried out at the second 

is already set, it is awaited until the communication stop flag phase However, as for as the intra--con:iputer IPC for the 

is reset. shared mcmofy, etc. is conocroed, it suffices to establish the 

Also, in the message passing, even when the system caU 30 synchronization within that computer, 

for the message transmission is executed, there is a case in For this reason, in this fifth embodiment, as shown in FIG. 

which the actual transmissian is delayed by the OS. Bxem- 21, each conqwiter is prorvided with an intra-computer syn- 

plary timings in this case are shown in FIGS. 19Aandl9B. chnmization unit 101, 102, etc for establishing die synchro- 

FIO. 19A shows a case in whidi the dieckpohiting is nixation of die processes within the computer. Then, at die 

started after die process A Issued the transmission system 55 first phase, the inter-NOomputer IPC is stopped, and at die 

caU, and die diedqpointing and die actual transmission are second phase, the local checkpointing is carried out after die 

canied out by the OS, while the process B carries out die iirtra-<xuiqnita' synchronization unit Is used to stop the 

cfaedqxjinting after die message is received. lo diis case, as intra^nqvuter IPC first By means of this, it is possible to 

shown in FIG. 19A, it appears as if there exists a **message further shorten the stopping time of the intra-coniputer IPC 

which is transmitted but not yet received". However, when 60 among the inter-process canununications. The first phase of 

a case of actually restarting frtmi the checkpoints is the fourth embodiment realizes the synchronization among 

considered, the processes restart fixon a state after the the computers, and this fifth eQibodiment iiHlt?^ the fact 

transmission system call is issued, so that it is going to be the tiut the intra-computer synchronization unit requires a 

same processing as that for t nm. smitting nt m timtng indicated shorter time compared widi the synchronization among die 

by a dashed line, and neither transmission nor reception is 6S computers. 

going to be carried out after the restart, and oonsequendy no Now, with references to the flow charts of FIG. 22, FIG. 

contradiction arises. 23 and FIG. 24, a procedure for the distributed checfepoint- 
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ing in tfus fifth cmbodiiMntwiU be described Here, FIG. 23 rality of processes v^cfa cany cut the inter-process cam- 
shows a flow chart of the operatioD by the ccmmunication municatioiis by the message passing or the mfarmation 
control unit 11, FIG. 23 shows a flow chart of the c^>eratioD exchange using shared memory or file sharing, while short- 
by the local checkpointing unit 12, and FIG. 24 shows a flow ening a stopping time of the processes due to the distributed 
cfaartof the operation by the intra-corx^Miter synchronization 5 checkpoindng. 

unit Here, the system management unit SM and the process It is to be noted that, besides those already mcntiotted 

management units PM are substantially similar to those used alxive, many modifications and variations of &e above 

in the fourth embodiment described above. embodiments may be made without departing from the 

The commumcation control unit 11 is awaiting for the novd and advantageous features of the present invention, 

communication control command (step S121). When the Accordin^y, all such modifications and variations are 

intcr-computcr IPC stop request is received (step S122 intended to be included within the scope of the ^jpendcd 

YES)t the inter-computer IPC is stepped (stq> S123). When daims. 

the intra-conqwtcr IPC stop request is received (step S125 What is claimed is: 

YES), the intta-coi^wto: IPC is iU^ppcd (step S126). When 1. A mc^od for checkpointing for a plurality of jHOcesses 

the communication resume request is received (step S127 <^>cfating under a distributed processing environment while 

YES), all the intcr-{xoccs$ communications are resumed carrying out int^-fvooess ccHimiunications, the meihod 

(step S128). When the processing in any of the steps S123, comiHisittg the 8tq>s of: 

S126 and S128 Is finished, a response to the request is stopping the inter-process communications of eadi pro- 
returned (step S124). cess while continuing a normal {oocessing of eadi 

The local checkpointing unit 12 is awaiting for the local ^ process; and 

chedqpoimlng request (step S131). When the local check- execudng a checkpointing processing for eadi process, 

pointing request is received, the local checkpointing unit 12 when the inter-process ccmmnnications of all fsocesses 

issues the intra-conqniter IPC stop request to the coanmu- are stc^iped at the stopping step, 

nicatioa control unit 11 (step S132). Then, the intra- 2. The method of claim 1, wherein the executing step also 

computer synchioniz^on unit is called up and the syndiro- ^ stops ttie nonmd processing of each process during flie 

nization within the oon^Miter is cairied out (step S153). checkpointing processing. 

Then, the stopping of the nosmal processing (stq> S134), the 3. The method of daim 2, further oon^rising the st^ of: 

local checkpointing (step S135), the resuming of the noimal resuming the inter-pcooess communications and the nor- 

processing (stq> S136), and the conmmnication resume mal processing of caA process when die c hff ck p ni nr i ng 

request to the communication control unit ll(stq>S137) are ^ processing for eadi process is finished, 

carricdout as in theprevious embodiments, and ttttrcyonse 4, The method of daim 1, further comprising the step 

is relumed to the process management unit (step S138). reooding dirty pages to which a writing is caxiied out by the 

Here, the processing of the local checkpointing unit 12 in normal processing of each process since a last chec^ioiiit; 

this fifth embodiment presupposes that the process is oper- wherein the executing step executes the checlqHTinting 

ating under the multi-thread environment Namely, die processing for staring only the dirty pages recorded at 

threads other duu the thread which is canying out the the recording st^ into a cfaeclqxsint, in paralld to the 

checkpointing are operating during the checkpointing, so noimal processing. 

that the intra^conq>uter IPC stop processing is carried out 5. ih^ noctfaod of daim 4, further coo^iising the stq> cf 

However, in a case ^(tae tike process is operating as a single stopping Hit noEtmal processing of each process when an 

direadfttieodierppocessinglntheprocesswillnotbecanied ^ access request to an unstored dirty page is issued die 

oat dming the checkpointing, so that tlie stopping of the normal processing of each process during the cfaedepointing 

intra-conqNiter IPC is unnecessary, and it suffices to simply processing for each process. 

estabUsb the synchronization tyy the intra-cuu^uilei syndiro- 6. The method of claim 5* ftinher cooopising the 8tq> of 

nization unit. resuming the normal processing for each process when the 

In the intra-conqjuter synchrocdzation unit, whether all 45 unstored dirty page is stored by the checkpointing process- 

the processes for carrying out the intra-con^tcr IPC art in ing. 

a waiting state or not is diecfced (step S141), and If so, all 7. The method of claim 4, wherein when an access request 

die waiting {xocesses are resumed (step $142% whoeas to an unstored dirty page is issued by the normal processing 

otherwise the Intra-conqMiter synchronization unit Is set in a daring die checkpointing processing, the chedqpolnting pro- 

waidng state (step S143). » oessing stores die unstored dhty page widi priority over 

To Illustrate die effect of diis fifth embodiment, FIO. 25A otha dirty pages, 

shows 8 case of die first to fourth «titwvitfiw>nf« descdbed 8. The method of dahoi 1, wherein die distributed pro- 

dbove, while FIG. 25B shows a case of this fiftti embodi- cessing system incbides a checkpoint mmmand process for 

menL In this fifth embodinient, only die inter-compater IPC controlling start and stopping of the checkpointing 

is stopped at the first phase, and the stopping of the intra- 35 processing, the method fMicr c omprisin g die stq^ of: 

computer IPC and the local checkpointing are carried out at sending an inter-process stop signal to said checkpoint 

die second phase. In addition. In the local diedqpointiDg command process firom each of said plurality of pn>- 

fsocessing, the storing of the dirty page is carried out in cesses upon stopping of ttie inter-process communicft- 

paralld to die normal processing aimilaily as in die second tions at said comsponding one of said plurality of 

and diird embodiments described above. 60 processes, wherein die nonnal processing of said each 

In contrast to a case <f die first to fourtti embodiments of said plurality of processes is continued until an 

shown in FIG. 25A whoc all the inter-process oommunica- intra^>roccss stop signal is received by said each of said 

tions are stopped at die first phase, it can be seen in FIG. 25B phiraUty of processes; and 

diat die stopping time of the intra-conq^uter IPC is further sending, &om said checkpoint command process to said 

shortened in this fifth embodinoeat 63 each of said plurality of processes, said intza-process 

As described, acccading to die present invention, it is stop signal when said inter^irocess stop signal has been 

possible to realize die distributed checkpointing for a plu- received from every one of said plurriity of processes. 
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9. A metijod for checkpamting for a plurality of pawesses 
opexatiiig under a distributed proccssiag environment while 
canylng out ioter-process communlcatLOOs, the method 
con^sing the steps of: 

c omm a n di n g each process to stop the intcr-process com- s 
munications of each process while continuiDg a normal 
processing of each process; and 

commanding each process to execute a checkpointing 
processing for each process, when a stopping of ihe 
inter-process conuounications of all processes is con- 
firmed. 

10. A method for dieckpointing for a plurality of pro- 
cesses operating under a distributed processing environment 
while carrying out inter-process communications, the 
method coiqvising the steps c£: 

stopping at eacb process the inter-process communica- 
tions of each process while continuing a aotmal pro- 
cessing of each process, upon receiving an inter- 
process communication stop command; and 

starting at cadi process an CKecution of a cfaedqpointing 
processing for each process, upon receiving a dieck- 
pointing mmmfliyl 

11. A mje^iod for chedcpcioting for a plurality d pro- 
cesses on coox^Hiters, the processes being (^>erating under ^ 
distributed processing environment while canying out inter- 
process communicatioas including inter-computer type 
inter-prooess oommunicadons to be canied out among the 
computers and intra-conqniter type inter-process communi- 
cations to be cauied cot within one cooiputer, the mediod ^ 
conqirising the steps of: 

stopping the inter-^^on^Niter type inter-inocess OHnmuni- 
cadons gI eadi process while oontiniiing the intra- 
computer type inter-process coaxummications and a 
noonal processing of eadi process; and 

executing a cbec^inting processing for each process 
while stopping the intra-oomputcr type Inter-process 
communications of eadi process, when the inter- 
CGBopatec type inter-process communications of aU 
processes axe stopped at ttie stopping stqx 40 

12. The method of claim U, wherein the executing step 
also stc^ the normal |sx>cessing of each process dming the 
dteckpdnting processing. 

13. The method of claim 12, further oon^risiiig the step 
of: 43 

resuming the inter-proccss communications and die nco^- 
mal processing of each process when the checkpointing 
processing for each process is flni^f^ 

14. The method of daim 11, further comprising the 8tq> 

of recording dirty pages to whidi a writing is caniod out by 50 
the normal processing of each process since a last dieck- 
point; 

herein die executing step executes the cfaeckpoindng 
processing for storing only the dirty pages recorded at 
the recording stq;> into a chedqpoint, in paralld to die 55 
normal processing. 

15. The method of daim 14, further oonqsrising the step 
of stopping the normal processing of eadi process when an 
access request to an unstoitd dirty page is issued by the 
normal processing of eadi process during the cfaedqpodnting tio 
processing for each process. 

16. The method of daim 15, finthcr oon^sing the step 
of resuming (he noonal ivooesslng for esdi process when 
the unstored ditty page is stored by the dieckpdndng 
processing. 65 

17. The method of daim 14, wherein when an access 
request to an unstoxed dirty page is issued by the normal 



I^ocessing during die dieckpointing processing, the check- 
pointing processing stores die unstcred dirty page with 
priority over oflier dirty pages. 

18. A method for checkpointing for a plurality of pro- 
cesses on computers, the processes being operating under a 
distributed processing environment while carrying out inter- 
process communications induding inter-computer type 
inter-process communications to be carried out among die 
computers and intra-computer type inter-proccss communi- 
cations to be canied out within one conqiutcr, the mcdiod 
comprising the steps of: 

commanding each process to stop the lnter-conq»uter type 
intcr-process commudcadons of eadi process wbilc 
conthiuing the intra-con^ter type inter-i)rooes5 com- 
munications and a normal processing of each process; 
and 

commanding each process to execute a diedqp(Hnting 
processing for eadi process while stopfong die intra- 
con^mter type inter-process communications of eadi 
isocess, when a stopping of die inter-oonqiuter type 
inter-process commimicatioos of all processes is con- 
firmed. 

19. A method for dieckpointing for a plurality of pro- 
cesses on computars, die processes being (q)erating under a 
distributed processing environment while carrying out inter- 
process communications induding intcr-computcr type 
inter-process communications to be carried out among die 
computers and intra-computer type inter-proccss communi- 
cations to be carried out within one computer, the method 
comprising the stq>s of: 

stopping at each process the inter-conqMiter type inter- 
process communications of each process while ccha- 
tindng die intra-oomputer type inter-process commu- 
nications and a normal processing of each process, 
upon receiving an iDter-computcr type Inter-process 
fi 'T m Tn u m'cadon stop command; ^ j t ^ 

starting at each process an execution of a checkpoontiag 
processing for eadi process while stc^qiing the intra- 
con^xiter type inter-process communicatioas of eadi 
process, upon receiving a checkpointing command. 

20. A computer system, comprising: 

a phmUity of process means operating under a distributed 
processing environment while carrying out inter- 
process communications; and 

dieckpointing command means for issuing an inter- 
process communication stop command r^nnpnwTPftjng a 
stopping of the inta-process commimlcations of eadi 
process means, and issuing a checkpointing command 
commanding an execution of a chedqpointing process- 
ing for each process means ^en responses to the 
inter-proccss coimmuication stop coEmmands £roai all 
process means are reodved; 

wherein each process means stops the inter-proccss com- 
munications of eadi process means wfaUe continuing a 
normal processing of eadi process means upon receiv- 
ing die inter-process communicatioa stop command 
and returns a response to the intcr-process communi- 
cation stop command when the intcr-process commu- 
nications of each process means are stopped, and starts 
an execution of the ched^xiinting processing for each 
process means upon receiving the dieckpointing com- 
mand. 

21. Ihe com|Hiter system of daim 20, further 

process management means, provided on each computer 
constituting the computer system, for managing die 
process means provided on eadi oon^wter; and 
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wherein die checkpointing command means is provided 
on one computer constituting the computer system, and 
issues the inter-pxocess communication stop command 
and the checkpointing command for e^ process 
means ttnoug^ the process management means which s 
manages each process means. 

2Z A computer systemi, comprising: 

a plurality of process means provided on computers, the 
process means t>eing opmting under a distributed 
processing environment while carrying out int»- 
process communications Including intcr-coaqwtcr type 
inter-process communications to be carried out among 
the computers and intxa-camputcr type inter-process 
communications to he carried out within one coixq>uter» 
and " 

dkockpointing coimnand means for issuing an inter- 
con^Nitcr type intcr-process commuidcation stop oomr 
mand commanding a stopjnng of the inter-computer 
type inter-proccss communicatioas of each process ^ 
means, and issuing a checkpointing oxnmand com^ 
manding an execution of a cfaedqMunting processing 



20 

for each process means and a stewing of the intra- 
counter type inter-process comiminicatioos of each 
process means when responses to the inter-con^tcr 
type inter-|HX)cess communication stop conmmnds 
from all process means are received; 
wherein each process means stops the inter-computer type 
inter-process communications of each process means 
wtifle continuing the intra-computcr type inter-process 
communications and a norma] processing of each pro- 
cess means upon receiving the inter-con^Miter type 
inter-ivocess conmuinication stop command and 
returns a response to the Intear-conqiuter type inter- 
process communication stop command v^cn the inter- 
computer type inter-process coTnmnni cations of each 
process means are stopped, and starts an execution of 
the checkpointing processing for each process means 
while stopping the intr»-conq>uter type inter-process 
communicatioas cf each process means upon receiving 
the checkpointing command. 

« « « « » 
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